{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# KFServing Deployment Pipeline For CIFAR10 Image Recognition\n",
    "\n",
    "In this example (and Kale Kubeflow pipeline) we create a deployment pipeline for a pretrained CIFAR10 image model. \n",
    "The following steps will be run:\n",
    "\n",
    " * Setup Minio client\n",
    " * Download and test CIFAR10 model and train an Anchors Images explainer on it. Save both model and explainer to Minio.\n",
    " * Deploy model and explainer using KFServing and test\n",
    " * Train outlier detector\n",
    " * Train drift detector\n",
    " * Deploy knative eventing display to show asynchronous results from outlier and drift detectors. \n",
    " * Deploy outlier detector and test\n",
    " * deploy drift detector and test\n",
    " \n",
    " ### Setup\n",
    " \n",
    " You will need a kubeflow cluster >= 1.0 with \n",
    "   \n",
    "  * Knative eventing \n",
    "  * Seldon >= 1.2.2\n",
    "  * KFServing >= 0.3.0\n",
    " \n",
    " ### Kubeflow Jupyter Notebook Server\n",
    " \n",
    " To run this notebook inside kubeflow. Create a Jupyter notebook server using the image `seldonio/jupyter-lab-alibi-kale:0.11`\n",
    " \n",
    " ### GCP Setup\n",
    " \n",
    " On GCP If you use Kale to save this notebook as a pipeline you will need to add the storage_class of the `VolumeOp` to `nfs-client` if you have followed the steps to create a NFS RWX PV on GCP. e.g.:\n",
    " \n",
    " ```\n",
    "    marshal_vop = dsl.VolumeOp(\n",
    "        name=\"kale-marshal-volume\",\n",
    "        resource_name=\"kale-marshal-pvc\",\n",
    "        storage_class=\"nfs-client\",\n",
    "        modes=dsl.VOLUME_MODE_RWM,\n",
    "        size=\"1Gi\"\n",
    "    )\n",
    " ```\n",
    " \n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "imports"
    ]
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "from alibi.explainers import AnchorImage\n",
    "from alibi.datasets import fetch_adult\n",
    "from minio import Minio\n",
    "from minio.error import ResponseError\n",
    "from joblib import dump, load\n",
    "import dill\n",
    "from subprocess import run, Popen, PIPE\n",
    "from alibi_detect.utils.data import create_outlier_batch\n",
    "from alibi_detect.utils.fetching import fetch_tf_model\n",
    "import json\n",
    "import logging\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "tf.keras.backend.clear_session()\n",
    "from tensorflow.keras.layers import Conv2D, Conv2DTranspose, Dense, Layer, Reshape, InputLayer\n",
    "from tqdm import tqdm\n",
    "\n",
    "from alibi_detect.models.losses import elbo\n",
    "from alibi_detect.od import OutlierVAE\n",
    "from alibi_detect.utils.fetching import fetch_detector\n",
    "from alibi_detect.utils.perturbation import apply_mask\n",
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "from alibi_detect.utils.visualize import plot_instance_score, plot_feature_outlier_image\n",
    "import time\n",
    "\n",
    "logger = tf.get_logger()\n",
    "logger.setLevel(logging.ERROR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Setup Pipeline Paramers\n",
    "\n",
    "The following global variables can be set. These will be used as Pipeline parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pipeline-parameters"
    ]
   },
   "outputs": [],
   "source": [
    "MINIO_HOST=\"minio-service.kubeflow:9000\"\n",
    "MINIO_ACCESS_KEY=\"minio\"\n",
    "MINIO_SECRET_KEY=\"minio123\"\n",
    "MINIO_MODEL_BUCKET=\"seldon\"\n",
    "CIFAR10_MODEL_PATH=\"tfserving/cifar10/model\"\n",
    "EXPLAINER_MODEL_PATH=\"tfserving/cifar10/explainer\"\n",
    "OUTLIER_MODEL_PATH=\"tfserving/cifar10/outlier\"\n",
    "DRIFT_MODEL_PATH=\"tfserving/cifar10/drift\"\n",
    "DEPLOY_NAMESPACE=\"admin\"\n",
    "TRAIN_OUTLIER_DETECTOR=False\n",
    "TRAIN_DRIFT_DETECTOR=False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "functions"
    ]
   },
   "outputs": [],
   "source": [
    "def get_minio():\n",
    "    return Minio(MINIO_HOST,\n",
    "                    access_key=MINIO_ACCESS_KEY,\n",
    "                    secret_key=MINIO_SECRET_KEY,\n",
    "                    secure=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:setup"
    ]
   },
   "outputs": [],
   "source": [
    "minioClient = get_minio()\n",
    "buckets = minioClient.list_buckets()\n",
    "for bucket in buckets:\n",
    "    print(bucket.name, bucket.creation_date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if not minioClient.bucket_exists(MINIO_MODEL_BUCKET):\n",
    "    minioClient.make_bucket(MINIO_MODEL_BUCKET)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Test and save Model\n",
    "\n",
    "For simplicity we will use a pretrained Resnet32 CIFAR10 tensorflow model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:train_model_and_explainer",
     "prev:setup"
    ]
   },
   "outputs": [],
   "source": [
    "model = fetch_tf_model('cifar10', 'resnet32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train, test = tf.keras.datasets.cifar10.load_data()\n",
    "X_train, y_train = train\n",
    "X_test, y_test = test\n",
    "\n",
    "X_train = X_train.astype('float32') / 255\n",
    "X_test = X_test.astype('float32') / 255\n",
    "print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',\n",
    "               'dog', 'frog', 'horse', 'ship', 'truck']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Test model locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "idx = 1\n",
    "X = X_test[idx].reshape(1, 32, 32, 3)\n",
    "plt.imshow(X.reshape(32, 32, 3))\n",
    "plt.axis('off')\n",
    "plt.show()\n",
    "print(\"class:\",class_names[y_test[idx][0]])\n",
    "print(\"prediction:\",class_names[model.predict(X_test[idx:idx+1])[0].argmax()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "modelfilepath=\"resnet\"\n",
    "tf.saved_model.save(model, modelfilepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "\n",
    "model_filepath=\"resnet\"\n",
    "print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{CIFAR10_MODEL_PATH}/1/saved_model.pb\", modelfilepath+\"/saved_model.pb\"))\n",
    "variable_filepath = modelfilepath+\"/variables\"\n",
    "onlyfiles = [f for f in listdir(variable_filepath) if isfile(join(variable_filepath, f))]\n",
    "for filename in onlyfiles:\n",
    "    print(filename)\n",
    "    print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{CIFAR10_MODEL_PATH}/1/variables/{filename}\", join(variable_filepath, filename)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train Explainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:"
    ]
   },
   "outputs": [],
   "source": [
    "def predict_fn(x):\n",
    "    return model.predict(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:"
    ]
   },
   "outputs": [],
   "source": [
    "\n",
    "image_shape = (32, 32, 3)\n",
    "segmentation_fn = 'slic'\n",
    "kwargs = {'n_segments': 5, 'compactness': 20, 'sigma': .5}\n",
    "explainer = AnchorImage(predict_fn, image_shape, segmentation_fn=segmentation_fn, \n",
    "                        segmentation_kwargs=kwargs, images_background=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "idx=0\n",
    "image = X_test[0]\n",
    "np.random.seed(0)\n",
    "explanation = explainer.explain(image, threshold=.95, p_sample=.5, tau=0.25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X = X_test[idx].reshape(1, 32, 32, 3)\n",
    "plt.imshow(X.reshape(32, 32, 3))\n",
    "plt.axis('off')\n",
    "plt.show()\n",
    "print(\"class:\",class_names[y_test[idx][0]])\n",
    "print(\"prediction:\",class_names[model.predict(X_test[idx:idx+1])[0].argmax()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "plt.imshow(explanation[\"anchor\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"explainer.dill\", \"wb\") as dill_file:\n",
    "    dill.dump(explainer, dill_file)    \n",
    "    dill_file.close()\n",
    "print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{EXPLAINER_MODEL_PATH}/explainer.dill\", 'explainer.dill'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train Outlier Detector\n",
    "\n",
    "For further details and extended notebook see [Alibi-Detect Documentation](https://docs.seldon.io/projects/alibi-detect/en/stable/). These steps were derived from [Alibi-Detect CIFAR10 Example](https://docs.seldon.io/projects/alibi-detect/en/stable/examples/od_vae_cifar10.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:train_outlier_detector",
     "prev:train_model_and_explainer"
    ]
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "tf.keras.backend.clear_session()\n",
    "from tensorflow.keras.layers import Conv2D, Conv2DTranspose, Dense, Layer, Reshape, InputLayer\n",
    "from tqdm import tqdm\n",
    "\n",
    "from alibi_detect.models.losses import elbo\n",
    "from alibi_detect.od import OutlierVAE\n",
    "from alibi_detect.utils.fetching import fetch_detector\n",
    "from alibi_detect.utils.perturbation import apply_mask\n",
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "from alibi_detect.utils.visualize import plot_instance_score, plot_feature_outlier_image\n",
    "\n",
    "logger = tf.get_logger()\n",
    "logger.setLevel(logging.ERROR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if TRAIN_OUTLIER_DETECTOR:\n",
    "    latent_dim = 1024\n",
    "    \n",
    "    encoder_net = tf.keras.Sequential(\n",
    "        [\n",
    "            InputLayer(input_shape=(32, 32, 3)),\n",
    "            Conv2D(64, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "            Conv2D(128, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "            Conv2D(512, 4, strides=2, padding='same', activation=tf.nn.relu)\n",
    "        ])\n",
    "\n",
    "    decoder_net = tf.keras.Sequential(\n",
    "        [\n",
    "            InputLayer(input_shape=(latent_dim,)),\n",
    "            Dense(4*4*128),\n",
    "            Reshape(target_shape=(4, 4, 128)),\n",
    "            Conv2DTranspose(256, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "            Conv2DTranspose(64, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "            Conv2DTranspose(3, 4, strides=2, padding='same', activation='sigmoid')\n",
    "        ])\n",
    "    \n",
    "     # initialize outlier detector\n",
    "    od = OutlierVAE(threshold=.015,  # threshold for outlier score\n",
    "                score_type='mse',  # use MSE of reconstruction error for outlier detection\n",
    "                encoder_net=encoder_net,  # can also pass VAE model instead\n",
    "                decoder_net=decoder_net,  # of separate encoder and decoder\n",
    "                latent_dim=latent_dim,\n",
    "                samples=2)\n",
    "    # train\n",
    "    od.fit(X_train, \n",
    "            loss_fn=elbo,\n",
    "            cov_elbo=dict(sim=.05),\n",
    "            epochs=50,\n",
    "            verbose=True)\n",
    "else:\n",
    "    od = load_detector(\"/home/models/samples/od/cifar10\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "idx = 8\n",
    "X = X_train[idx].reshape(1, 32, 32, 3)\n",
    "X_recon = od.vae(X)\n",
    "plt.imshow(X.reshape(32, 32, 3))\n",
    "plt.axis('off')\n",
    "plt.show()\n",
    "plt.imshow(X_recon.numpy().reshape(32, 32, 3))\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X = X_train[:500]\n",
    "print(X.shape)\n",
    "od_preds = od.predict(X,\n",
    "                      outlier_type='instance',    # use 'feature' or 'instance' level\n",
    "                      return_feature_score=True,  # scores used to determine outliers\n",
    "                      return_instance_score=True)\n",
    "print(list(od_preds['data'].keys()))\n",
    "target = np.zeros(X.shape[0],).astype(int)  # all normal CIFAR10 training instances\n",
    "labels = ['normal', 'outlier']\n",
    "plot_instance_score(od_preds, target, labels, od.threshold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "\n",
    "filepath=\"cifar10outlier\"\n",
    "save_detector(od, filepath) \n",
    "onlyfiles = [f for f in listdir(filepath) if isfile(join(filepath, f))]\n",
    "for filename in onlyfiles:\n",
    "    print(filename)\n",
    "    print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{OUTLIER_MODEL_PATH}/{filename}\", join(filepath, filename)))\n",
    "filepath=\"cifar10outlier/model\"\n",
    "onlyfiles = [f for f in listdir(filepath) if isfile(join(filepath, f))]\n",
    "for filename in onlyfiles:\n",
    "    print(filename)\n",
    "    print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{OUTLIER_MODEL_PATH}/model/{filename}\", join(filepath, filename)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train a Drift Detector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:train_drift_detector",
     "prev:train_model_and_explainer"
    ]
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import os\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Conv2D, Dense, Flatten, InputLayer, Reshape\n",
    "\n",
    "from alibi_detect.cd import KSDrift\n",
    "from alibi_detect.cd.preprocess import uae, hidden_output\n",
    "from alibi_detect.models.resnet import scale_by_instance\n",
    "from alibi_detect.utils.fetching import fetch_tf_model, fetch_detector\n",
    "from alibi_detect.utils.prediction import predict_batch\n",
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "from alibi_detect.datasets import fetch_cifar10c, corruption_types_cifar10c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tf.random.set_seed(0)\n",
    "\n",
    "if True:\n",
    "    np.random.seed(0)\n",
    "    n_test = X_test.shape[0]\n",
    "    idx = np.random.choice(n_test, size=n_test // 2, replace=False)\n",
    "    idx_h0 = np.delete(np.arange(n_test), idx, axis=0)\n",
    "    X_ref,y_ref = X_test[idx], y_test[idx]\n",
    "    X_h0, y_h0 = X_test[idx_h0], y_test[idx_h0]\n",
    "    print(X_ref.shape, X_h0.shape)\n",
    "    # define encoder\n",
    "    encoding_dim = 32\n",
    "    encoder_net = tf.keras.Sequential(\n",
    "      [\n",
    "          InputLayer(input_shape=(32, 32, 3)),\n",
    "          Conv2D(64, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "          Conv2D(128, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "          Conv2D(512, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "          Flatten(),\n",
    "          Dense(encoding_dim,)\n",
    "      ]\n",
    "    )\n",
    "\n",
    "    # initialise drift detector\n",
    "    p_val = .05\n",
    "    cd = KSDrift(\n",
    "        p_val=p_val,        # p-value for K-S test\n",
    "        X_ref=X_ref,       # test against original test set\n",
    "        preprocess_fn=uae,  # UAE for dimensionality reduction\n",
    "        preprocess_kwargs={'encoder_net': encoder_net, 'batch_size': 128},\n",
    "        alternative='two-sided'  # other options: 'less', 'greater'\n",
    "    )\n",
    "else:\n",
    "    cd = load_detector(\"/home/models/samples/cd/cifar10\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "\n",
    "filepath=\"cifar10Drift\"\n",
    "save_detector(cd, filepath) \n",
    "onlyfiles = [f for f in listdir(filepath) if isfile(join(filepath, f))]\n",
    "for filename in onlyfiles:\n",
    "    print(filename)\n",
    "    print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{DRIFT_MODEL_PATH}/{filename}\", join(filepath, filename)))\n",
    "filepath=\"cifar10Drift/model\"\n",
    "onlyfiles = [f for f in listdir(filepath) if isfile(join(filepath, f))]\n",
    "for filename in onlyfiles:\n",
    "    print(filename)\n",
    "    print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{DRIFT_MODEL_PATH}/model/{filename}\", join(filepath, filename)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy KFServing Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:deploy_model",
     "prev:train_model_and_explainer"
    ]
   },
   "outputs": [],
   "source": [
    "secret=f\"\"\"apiVersion: v1\n",
    "kind: Secret\n",
    "metadata:\n",
    "  name: cifar10-kf-secret\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "  annotations:\n",
    "     serving.kubeflow.org/s3-endpoint: {MINIO_HOST} # replace with your s3 endpoint\n",
    "     serving.kubeflow.org/s3-usehttps: \"0\" # by default 1, for testing with minio you need to set to 0\n",
    "type: Opaque\n",
    "stringData:\n",
    "  awsAccessKeyID: {MINIO_ACCESS_KEY}\n",
    "  awsSecretAccessKey: {MINIO_SECRET_KEY}\n",
    "\"\"\"\n",
    "with open(\"secret.yaml\",\"w\") as f:\n",
    "    f.write(secret)\n",
    "run(\"kubectl apply -f secret.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "secret = f\"\"\"apiVersion: v1\n",
    "kind: Secret\n",
    "metadata:\n",
    "  name: seldon-init-container-secret\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "type: Opaque\n",
    "stringData:\n",
    "  AWS_ACCESS_KEY_ID: {MINIO_ACCESS_KEY}\n",
    "  AWS_SECRET_ACCESS_KEY: {MINIO_SECRET_KEY}\n",
    "  AWS_ENDPOINT_URL: http://{MINIO_HOST}\n",
    "  USE_SSL: \"false\"\n",
    "\"\"\"\n",
    "with open(\"secret.yaml\",\"w\") as f:\n",
    "    f.write(secret)\n",
    "run(\"cat secret.yaml | kubectl apply -f -\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sa = f\"\"\"apiVersion: v1\n",
    "kind: ServiceAccount\n",
    "metadata:\n",
    "  name: minio-kf-sa\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "secrets:\n",
    "  - name: cifar10-kf-secret\n",
    "\"\"\"\n",
    "with open(\"sa.yaml\",\"w\") as f:\n",
    "    f.write(sa)\n",
    "run(\"kubectl apply -f sa.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from kubernetes import client\n",
    "from kfserving import KFServingClient\n",
    "from kfserving import constants\n",
    "from kfserving import utils\n",
    "from kfserving import V1alpha2EndpointSpec\n",
    "from kfserving import V1alpha2PredictorSpec\n",
    "from kfserving import V1alpha2ExplainerSpec\n",
    "from kfserving import V1alpha2AlibiExplainerSpec\n",
    "from kfserving import V1alpha2TensorflowSpec\n",
    "from kfserving import V1alpha2InferenceServiceSpec\n",
    "from kfserving import V1alpha2InferenceService\n",
    "from kfserving import V1alpha2Logger\n",
    "from kubernetes.client import V1ResourceRequirements\n",
    "\n",
    "api_version = constants.KFSERVING_GROUP + '/' + constants.KFSERVING_VERSION\n",
    "default_endpoint_spec = V1alpha2EndpointSpec(\n",
    "                          predictor=V1alpha2PredictorSpec(\n",
    "                            service_account_name='minio-kf-sa',\n",
    "                            tensorflow=V1alpha2TensorflowSpec(\n",
    "                              storage_uri='s3://'+MINIO_MODEL_BUCKET+'/'+ CIFAR10_MODEL_PATH,\n",
    "                              resources=V1ResourceRequirements(\n",
    "                                  requests={'cpu':'100m','memory':'1Gi'},\n",
    "                                  limits={'cpu':'100m', 'memory':'1Gi'})),\n",
    "                            logger=V1alpha2Logger(\n",
    "                                mode='all'\n",
    "                            )),\n",
    "                            explainer=V1alpha2ExplainerSpec(\n",
    "                              service_account_name='minio-kf-sa',\n",
    "                            alibi=V1alpha2AlibiExplainerSpec(\n",
    "                              type='AnchorImages',\n",
    "                              storage_uri='s3://'+MINIO_MODEL_BUCKET+'/'+ EXPLAINER_MODEL_PATH,\n",
    "                              resources=V1ResourceRequirements(\n",
    "                                  requests={'cpu':'100m','memory':'1Gi'},\n",
    "                                  limits={'cpu':'100m', 'memory':'1Gi'}))))\n",
    "    \n",
    "isvc = V1alpha2InferenceService(api_version=api_version,\n",
    "                          kind=constants.KFSERVING_KIND,\n",
    "                          metadata=client.V1ObjectMeta(\n",
    "                              name='kf-cifar10', namespace=DEPLOY_NAMESPACE),\n",
    "                          spec=V1alpha2InferenceServiceSpec(default=default_endpoint_spec))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "KFServing = KFServingClient()\n",
    "KFServing.create(isvc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "KFServing.get('kf-cifar10', namespace=DEPLOY_NAMESPACE, watch=True, timeout_seconds=240)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Test Model and Explainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:test_model_and_explainer",
     "prev:deploy_model"
    ]
   },
   "outputs": [],
   "source": [
    "def test_model():\n",
    " idx=10\n",
    " test_example=X_test[idx:idx+1].tolist()\n",
    " payload='{\"instances\":'+f\"{test_example}\"+' }'\n",
    " cmd=f\"\"\"curl -v -d '{payload}' \\\n",
    "   -H \"Host: kf-cifar10.admin.example.com\" \\\n",
    "   -H \"Content-Type: application/json\" \\\n",
    "   http://kfserving-ingressgateway.istio-system/v1/models/kf-cifar10:predict\n",
    " \"\"\"\n",
    " ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    " raw = ret.stdout.read().decode(\"utf-8\")\n",
    " print(raw)\n",
    " res=json.loads(raw)\n",
    " arr=np.array(res[\"predictions\"])\n",
    " X = X_test[idx].reshape(1, 32, 32, 3)\n",
    " plt.imshow(X.reshape(32, 32, 3))\n",
    " plt.axis('off')\n",
    " plt.show()\n",
    " print(\"class:\",class_names[y_test[idx][0]])\n",
    " print(\"prediction:\",class_names[arr[0].argmax()])\n",
    "\n",
    "ok = False\n",
    "while not ok:\n",
    "    try:\n",
    "        test_model()\n",
    "        ok = True\n",
    "    except:\n",
    "        print(\"Failed calling model, sleeping\")\n",
    "        time.sleep(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Make an explanation request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "idx=1\n",
    "test_example=X_test[idx:idx+1].tolist()\n",
    "payload='{\"instances\":'+f\"{test_example}\"+' }'\n",
    "cmd=f\"\"\"curl -v -d '{payload}' \\\n",
    "   -H \"Host: kf-cifar10.admin.example.com\" \\\n",
    "   -H \"Content-Type: application/json\" \\\n",
    "   http://kfserving-ingressgateway.istio-system/v1/models/kf-cifar10:explain\n",
    "\"\"\"\n",
    "ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "raw = ret.stdout.read().decode(\"utf-8\")\n",
    "res=json.loads(raw)\n",
    "plt.imshow(np.array(explanation[\"anchor\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy KNative Eventing Event Display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:deploy_event_display",
     "prev:train_drift_detector",
     "prev:train_outlier_detector",
     "prev:test_model_and_explainer"
    ]
   },
   "outputs": [],
   "source": [
    "event_display=f\"\"\"apiVersion: apps/v1\n",
    "kind: Deployment\n",
    "metadata:\n",
    "  name: event-display\n",
    "  namespace: {DEPLOY_NAMESPACE}          \n",
    "spec:\n",
    "  replicas: 1\n",
    "  selector:\n",
    "    matchLabels: &labels\n",
    "      app: event-display\n",
    "  template:\n",
    "    metadata:\n",
    "      labels: *labels\n",
    "    spec:\n",
    "      containers:\n",
    "        - name: helloworld-go\n",
    "          # Source code: https://github.com/knative/eventing-contrib/tree/master/cmd/event_display\n",
    "          image: gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display@sha256:f4628e97a836c77ed38bd3b6fd3d0b06de4d5e7db6704772fe674d48b20bd477\n",
    "---\n",
    "kind: Service\n",
    "apiVersion: v1\n",
    "metadata:\n",
    "  name: event-display\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  selector:\n",
    "    app: event-display\n",
    "  ports:\n",
    "    - protocol: TCP\n",
    "      port: 80\n",
    "      targetPort: 8080\n",
    "---\n",
    "apiVersion: eventing.knative.dev/v1alpha1\n",
    "kind: Trigger\n",
    "metadata:\n",
    "  name: cifar10-outlier-display\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  broker: default\n",
    "  filter:\n",
    "    attributes:\n",
    "      type: org.kubeflow.serving.inference.outlier\n",
    "  subscriber:\n",
    "    ref:\n",
    "      apiVersion: v1\n",
    "      kind: Service\n",
    "      name: event-display\n",
    "---\n",
    "apiVersion: eventing.knative.dev/v1alpha1\n",
    "kind: Trigger\n",
    "metadata:\n",
    "  name: cifar10-drift-display\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  broker: default\n",
    "  filter:\n",
    "    attributes:\n",
    "      type: org.kubeflow.serving.inference.drift\n",
    "  subscriber:\n",
    "    ref:\n",
    "      apiVersion: v1\n",
    "      kind: Service\n",
    "      name: event-display\n",
    "\"\"\"\n",
    "with open(\"event_display.yaml\",\"w\") as f:\n",
    "    f.write(event_display)\n",
    "run(\"kubectl apply -f event_display.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl rollout status -n {DEPLOY_NAMESPACE} deploy/event-display -n {DEPLOY_NAMESPACE}\", shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy KFServing Outlier Detector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:deploy_outlier_detector",
     "prev:deploy_event_display"
    ]
   },
   "outputs": [],
   "source": [
    "outlier_yaml=f\"\"\"apiVersion: serving.knative.dev/v1\n",
    "kind: Service\n",
    "metadata:\n",
    "  name: cifar10-outlier\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  template:\n",
    "    metadata:\n",
    "      annotations:\n",
    "        autoscaling.knative.dev/minScale: \"1\"\n",
    "    spec:\n",
    "      containers:\n",
    "      - image: seldonio/alibi-detect-server:1.2.1\n",
    "        imagePullPolicy: IfNotPresent\n",
    "        args:\n",
    "        - --model_name\n",
    "        - cifar10od\n",
    "        - --protocol\n",
    "        - tensorflow.http\n",
    "        - --storage_uri\n",
    "        - s3://{MINIO_MODEL_BUCKET}/{OUTLIER_MODEL_PATH}\n",
    "        - --reply_url\n",
    "        - http://default-broker       \n",
    "        - --event_type\n",
    "        - org.kubeflow.serving.inference.outlier\n",
    "        - --event_source\n",
    "        - org.kubeflow.serving.cifar10od\n",
    "        - OutlierDetector\n",
    "        envFrom:\n",
    "        - secretRef:\n",
    "            name: seldon-init-container-secret\n",
    "\"\"\"\n",
    "with open(\"outlier.yaml\",\"w\") as f:\n",
    "    f.write(outlier_yaml)\n",
    "run(\"kubectl apply -f outlier.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "trigger_outlier_yaml=f\"\"\"apiVersion: eventing.knative.dev/v1alpha1\n",
    "kind: Trigger\n",
    "metadata:\n",
    "  name: cifar10-outlier-trigger\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  filter:\n",
    "    sourceAndType:\n",
    "      type: org.kubeflow.serving.inference.request\n",
    "  subscriber:\n",
    "    ref:\n",
    "      apiVersion: serving.knative.dev/v1\n",
    "      kind: Service\n",
    "      name: cifar10-outlier\n",
    "\"\"\"\n",
    "with open(\"outlier_trigger.yaml\",\"w\") as f:\n",
    "    f.write(trigger_outlier_yaml)\n",
    "run(\"kubectl apply -f outlier_trigger.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl rollout status -n {DEPLOY_NAMESPACE} deploy/$(kubectl get deploy -l serving.knative.dev/service=cifar10-outlier -o jsonpath='{{.items[0].metadata.name}}' -n {DEPLOY_NAMESPACE})\", shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Test KFServing Outlier Detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:test_oulier_detection",
     "prev:deploy_outlier_detector"
    ]
   },
   "outputs": [],
   "source": [
    "idx = 1\n",
    "X = X_train[idx:idx+1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "np.random.seed(0) \n",
    "X_mask, mask = apply_mask(X.reshape(1, 32, 32, 3),\n",
    "                                  mask_size=(10,10),\n",
    "                                  n_masks=1,\n",
    "                                  channels=[0,1,2],\n",
    "                                  mask_type='normal',\n",
    "                                  noise_distr=(0,1),\n",
    "                                  clip_rng=(0,1))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def predict():\n",
    "    test_example=X_mask.tolist()\n",
    "    payload='{\"instances\":'+f\"{test_example}\"+' }'\n",
    "    cmd=f\"\"\"curl -v -d '{payload}' \\\n",
    "       -H \"Host: kf-cifar10.admin.example.com\" \\\n",
    "       -H \"Content-Type: application/json\" \\\n",
    "       http://kfserving-ingressgateway.istio-system/v1/models/kf-cifar10:predict\n",
    "     \"\"\"\n",
    "    ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "    raw = ret.stdout.read().decode(\"utf-8\")\n",
    "    print(raw)\n",
    "    res=json.loads(raw)\n",
    "    arr=np.array(res[\"predictions\"])\n",
    "    plt.imshow(X_mask.reshape(32, 32, 3))\n",
    "    plt.axis('off')\n",
    "    plt.show()\n",
    "    print(\"class:\",class_names[y_train[idx][0]])\n",
    "    print(\"prediction:\",class_names[arr[0].argmax()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_outlier_event_display_logs():\n",
    "    cmd=f\"kubectl logs $(kubectl get pod -l app=event-display -o jsonpath='{{.items[0].metadata.name}}' -n {DEPLOY_NAMESPACE}) -n {DEPLOY_NAMESPACE}\"\n",
    "    ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "    res = ret.stdout.read().decode(\"utf-8\").split(\"\\n\")\n",
    "    data= []\n",
    "    for i in range(0,len(res)):\n",
    "        if res[i] == 'Data,':\n",
    "            j = json.loads(json.loads(res[i+1]))\n",
    "            if \"is_outlier\"in j[\"data\"].keys():\n",
    "                data.append(j)\n",
    "    if len(data) > 0:\n",
    "        return data[-1]\n",
    "    else:\n",
    "        return None\n",
    "j = None\n",
    "while j is None:\n",
    "    predict()\n",
    "    print(\"Waiting for outlier logs, sleeping\")\n",
    "    time.sleep(2)\n",
    "    j = get_outlier_event_display_logs()\n",
    "    \n",
    "print(j)\n",
    "print(\"Outlier\",j[\"data\"][\"is_outlier\"]==[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy KFServing Drift Detector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:deploy_drift_detector",
     "prev:test_oulier_detection"
    ]
   },
   "outputs": [],
   "source": [
    "drift_yaml=f\"\"\"apiVersion: serving.knative.dev/v1\n",
    "kind: Service\n",
    "metadata:\n",
    "  name: cifar10-drift\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  template:\n",
    "    metadata:\n",
    "      annotations:\n",
    "        autoscaling.knative.dev/minScale: \"1\"\n",
    "    spec:\n",
    "      containers:\n",
    "      - image: seldonio/alibi-detect-server:1.2.2-dev\n",
    "        imagePullPolicy: IfNotPresent\n",
    "        args:\n",
    "        - --model_name\n",
    "        - cifar10cd\n",
    "        - --protocol\n",
    "        - tensorflow.http\n",
    "        - --storage_uri\n",
    "        - s3://{MINIO_MODEL_BUCKET}/{DRIFT_MODEL_PATH}\n",
    "        - --reply_url\n",
    "        - http://default-broker\n",
    "        - --event_type\n",
    "        - org.kubeflow.serving.inference.drift\n",
    "        - --event_source\n",
    "        - org.kubeflow.serving.cifar10cd\n",
    "        - DriftDetector\n",
    "        - --drift_batch_size\n",
    "        - '500'\n",
    "        envFrom:\n",
    "        - secretRef:\n",
    "            name: seldon-init-container-secret\n",
    "\"\"\"\n",
    "with open(\"drift.yaml\",\"w\") as f:\n",
    "    f.write(drift_yaml)\n",
    "run(\"kubectl apply -f drift.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "trigger_outlier_yaml=f\"\"\"apiVersion: eventing.knative.dev/v1alpha1\n",
    "kind: Trigger\n",
    "metadata:\n",
    "  name: cifar10-drift-trigger\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  filter:\n",
    "    sourceAndType:\n",
    "      type: org.kubeflow.serving.inference.request\n",
    "  subscriber:\n",
    "    ref:\n",
    "      apiVersion: serving.knative.dev/v1\n",
    "      kind: Service\n",
    "      name: cifar10-drift\n",
    "\"\"\"\n",
    "with open(\"outlier_trigger.yaml\",\"w\") as f:\n",
    "    f.write(trigger_outlier_yaml)\n",
    "run(\"kubectl apply -f outlier_trigger.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl rollout status -n {DEPLOY_NAMESPACE} deploy/$(kubectl get deploy -l serving.knative.dev/service=cifar10-drift -o jsonpath='{{.items[0].metadata.name}}' -n {DEPLOY_NAMESPACE})\", shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Test KFServing Drift Detector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:test_drift_detector",
     "prev:deploy_drift_detector"
    ]
   },
   "outputs": [],
   "source": [
    "def show(X):\n",
    "    plt.imshow(X.reshape(32, 32, 3))\n",
    "    plt.axis('off')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from alibi_detect.datasets import fetch_cifar10c, corruption_types_cifar10c\n",
    "corruption = ['motion_blur']\n",
    "X_corr, y_corr = fetch_cifar10c(corruption=corruption, severity=5, return_X_y=True)\n",
    "X_corr = X_corr.astype('float32') / 255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "show(X_corr[0])\n",
    "show(X_corr[1])\n",
    "show(X_corr[2])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def predict(X):\n",
    "    test_example=X.tolist()\n",
    "    payload='{\"instances\":'+f\"{test_example}\"+' }'\n",
    "    with open(\"payload.json\",\"w\") as f:\n",
    "        f.write(payload)\n",
    "    cmd=f\"\"\"curl -d @./payload.json \\\n",
    "       -H \"Host: kf-cifar10.admin.example.com\" \\\n",
    "       -H \"Content-Type: application/json\" \\\n",
    "       http://kfserving-ingressgateway.istio-system/v1/models/kf-cifar10:predict\n",
    "    \"\"\"\n",
    "    run(cmd, shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_drift_event_display_logs():\n",
    "    cmd=f\"kubectl logs $(kubectl get pod -l app=event-display -o jsonpath='{{.items[0].metadata.name}}' -n {DEPLOY_NAMESPACE}) -n {DEPLOY_NAMESPACE}\"\n",
    "    ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "    res = ret.stdout.read().decode(\"utf-8\").split(\"\\n\")\n",
    "    data= []\n",
    "    for i in range(0,len(res)):\n",
    "        if res[i] == 'Data,':\n",
    "            j = json.loads(json.loads(res[i+1]))\n",
    "            if \"is_drift\" in j[\"data\"].keys():\n",
    "                data.append(j)\n",
    "    if len(data) > 0:\n",
    "        return data[-1]\n",
    "    else:\n",
    "        return None\n",
    "j = None\n",
    "for i in range(0,1000,50):\n",
    "    X = X_corr[i:i+50]\n",
    "    predict(X)\n",
    "    print(\"Waiting for drift logs, sleeping\")\n",
    "    time.sleep(2)\n",
    "    j = get_drift_event_display_logs()\n",
    "    if j is not None:\n",
    "        break\n",
    "    \n",
    "print(j)\n",
    "print(\"Drift\",j[\"data\"][\"is_drift\"]==1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Clean up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "skip"
    ]
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl delete inferenceservice kf-cifar10 -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete ksvc cifar10-outlier -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete ksvc cifar10-drift -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete trigger --all -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete sa  minio-kf-sa -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete secret seldon-init-container-secret -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete secret cifar10-kf-secret -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete deployment event-display -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete svc event-display -n {DEPLOY_NAMESPACE}\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "kubeflow_notebook": {
   "docker_image": "seldonio/jupyter-lab-alibi-kale:0.11",
   "experiment": {
    "id": "new",
    "name": "kfserving-e2e-cifar10"
   },
   "experiment_name": "kfserving-e2e-cifar10",
   "katib_metadata": {
    "algorithm": {
     "algorithmName": "grid"
    },
    "maxFailedTrialCount": 3,
    "maxTrialCount": 12,
    "objective": {
     "objectiveMetricName": "",
     "type": "minimize"
    },
    "parallelTrialCount": 3,
    "parameters": []
   },
   "katib_run": false,
   "pipeline_description": "KFServing CIFAR10 Example",
   "pipeline_name": "kfserving-e2e-cifar10",
   "volumes": []
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
